Shape refreshment need metric for object-based resilient video coding

Although there are several techniques that video encoders may use to improve error resilience, it is largely recognized that intra coding refreshment plays a major role. This technique is especially useful for video encoders that rely on predictive (inter) coding to remove temporal redundancy because, in these conditions, the decoded quality can decay very rapidly due to error propagation if errors occur in the transmission or storage of the coded streams. Therefore, in order to avoid error propagation for too long a time, the encoder can use a coding refreshment scheme to refresh the decoding process and stop (spatial and temporal) error propagation. In the context of object-based video coding, the video encoder can apply intra coding refreshment to both the shape and the texture data. In this paper, a shape refreshment need metric is proposed which can be used by object-based video encoders, notably MPEG-4 video encoders, to determine when the shape of a given video object should be refreshed in order to improve the decoded video quality. 1. CONTEXT AND OBJECTIVES With the availability of the MPEG-4 object-based audiovisual coding standard [1], new multimedia services and devices are making their way into the market. In particular, there is an increasing interest in mobile multimedia services, such as mobile videotelephony and mobile video streaming. However, mobile networks are highly error-prone environments, which makes it difficult, if not impossible, to transmit highly compressed video data with an acceptable quality without appropriate error resilience techniques. Error resilience techniques, and especially error concealment, are usually seen as playing a role at the decoder side of the communication chain. However, this is only partly true since the encoder and the bitstream syntax itself play an important role on what can be achieved at the decoder side in terms of error processing. For instance, it is largely recognized that intra coding refreshment can play a major role in improving error resilience in video coding systems that rely on predictive (inter) coding to remove temporal redundancy because, in these conditions, the decoded quality can decay very rapidly due to error propagation if errors occur in the transmission or storage of the coded streams. Therefore, in order to avoid error propagation for too long a time, the encoder can use an intra coding refreshment scheme to refresh the decoding process and stop (spatial and temporal) error propagation. The refreshment process will certainly decrease the coding efficiency, but it will significantly improve error resilience at the decoder side, increasing the overall subjective impact. In order to design an efficient intra coding refreshment scheme for an object-based coding scheme, it would be helpful for the encoder to have a method to determine which parts of the video data (shape and texture) should be refreshed and when, i.e. a refreshment need measure. The refreshment need for a given visual data entity, such as a shape or a texture macroblock for macroblock-based coding schemes, is a metric that measures the necessity of refreshing this visual data entity according to some error resilience criteria and can be used by the encoder to decide if a given visual entity should be refreshed or not at a certain point in time. In the following, a shape refreshment need (SRN) metric is proposed; this metric is to be integrated in a complete intra coding refreshment scheme, considering shape and texture. 2. DEFINING SHAPE REFRESHMENT NEED (SRN) In MPEG-4 shape coding [1], the only way to eliminate the shape dependency on the past is by refreshing the entire shape of a video object plane (VOP). Therefore, although shape coding is blockbased, the SRN measure should be VOP-based. This way, by considering the most important factors impacting the SRN, the SRNi parameter that measures the SRN for the i-th VOP in a video object sequence, can be defined as: ( ) i i SCD i i TSCD SSCD f SEV SRN , × = (1) where SEVi is the Shape Error Vulnerability (SEV) for VOP i, i SSCD is the normalized Spatial Shape Concealment Difficulty (SSCD) for VOP i (with values between 0 and 1), i TSCD is the normalized Temporal Shape Concealment Difficulty (TSCD) for VOP i (with values between 0 and 1), and fSCD is a function that combines the values of SSCD and TSCD into a single Shape Concealment Difficulty (SCD) value. SRNi is defined as a product where the first factor is the SEV measuring the statistical exposure of the shape data to errors, already considering the used bitstream structure, and the second factor corresponds to the SCD which expresses how hard the shape is to recover using concealment techniques. A product is used because the dependency of the SRN on the SEV and the SCD should be such that when either one of them approaches zero, so should the SRN. For instance, if the considered shape is extremely easy to conceal (SCD approaching zero), the SRN should also approach zero independently of the SEV. Or, on the other hand, if the shape is not vulnerable to errors (because almost no bits were spent or the amount of errors is very small), the SRN should also approach zero independently of the SCD. 2.1 Defining Shape Error Vulnerability It is proposed here that the first factor in Equation (1) be measured by the fraction of shape bits that will have to be discarded due to errors in VOP i. Since in MPEG-4, each VOP is divided in several video packets (VP), SEVi becomes: